System and method for determining user interactions with visual content presented in a mixed reality environment

ABSTRACT

Embodiments of the present invention provide a system for generating and displaying tailored advertisements in a mixed reality environment. The system is configured for continuously identifying one or more objects in a mixed reality environment, identifying the one or more objects match advertising targets, generating a tailored advertisement, transmitting the tailored advertisement to the user device, causing the user device to display the tailored advertisement, determining that the user is interacting with the tailored advertisement, and capturing one or more metrics associated with the interaction of the user with the tailored advertisement.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of co-pending U.S. patentapplication Ser. No. 17/007,429, of the same title and filed on Aug. 31,2020, which is assigned to the assignee hereof and the contents thereofare hereby expressly incorporated by reference.

FIELD OF THE INVENTION

The present invention relates to mixed reality environments and, morespecifically, to generating and displaying tailored advertisements inmixed reality environments.

BACKGROUND

Existing conventional systems typically focus on effectively deliveringadvertisements in websites, social media platforms, or the like.However, with increase in technology associated with mixed reality, theexisting methods of delivering and converting advertisements cannot beapplied to mixed reality environments. As such, there exists a need fora system to generate and display advertisements in mixed realityenvironments.

BRIEF SUMMARY

The following presents a summary of certain embodiments of theinvention. This summary is not intended to identify key or criticalelements of all embodiments nor delineate the scope of any or allembodiments. Its sole purpose is to present certain concepts andelements of one or more embodiments in a summary form as a prelude tothe more detailed description that follows.

Embodiments of the present invention address the above needs and/orachieve other advantages by providing apparatuses (e.g., a system,computer program product and/or other devices) and methods forgenerating and displaying advertisements in a mixed reality environment.The system embodiments may comprise one or more memory devices havingcomputer readable program code stored thereon, a communication device,and one or more processing devices operatively coupled to the one ormore memory devices, wherein the one or more processing devices areconfigured to execute the computer readable program code to carry outthe invention. In computer program product embodiments of the invention,the computer program product comprises at least one non-transitorycomputer readable medium comprising computer readable instructions forcarrying out the invention. Computer implemented method embodiments ofthe invention may comprise providing a computing system comprising acomputer processing device and a non-transitory computer readablemedium, where the computer readable medium comprises configured computerprogram instruction code, such that when said instruction code isoperated by said computer processing device, said computer processingdevice performs certain operations to carry out the invention.

In some embodiments, the present invention displays visual contentassociated with an advertisement in a mixed reality environment of auser, via a user device of the user, in response to displaying thevisual content, perform at least one of one or more actions ofcontinuously monitoring gaze of the user via the user device,continuously monitoring one or more gestures of the user, via the userdevice, and continuously monitoring one or more expressions of the user,via the user device, determine that the user is interacting with thevisual content based on at least one of the gaze, the one or moregestures, and the one or more expressions of the user.

In some embodiments, the present invention generates and displaysadditional visual content in response to determining that the user isinteracting with the visual content.

In some embodiments, the present invention performs at least one of theone or more actions based on determining type of the mixed realityenvironment of the user and determining type of the visual content.

In some embodiments, the type of visual content is at least a twodimensional advertisement or a three dimensional advertisement, whereinthe two dimensional advertisement comprises a two dimensional image or atwo dimensional video, wherein the three dimensional advertisementcomprises at least one of a three dimensional animation, a threedimensional image, a three dimensional video.

In some embodiments, the type of the mixed reality environment is avirtual reality environment. In some embodiments, the present inventionthe type of the mixed reality environment is an augmented realityenvironment.

In some embodiments, the present invention determines that the user isinteracting with the visual content based on matching the at least oneof the gaze, the one or more gestures, and the one or more expressionsof the user with one or more interactions gestures associated with thevisual content.

The features, functions, and advantages that have been discussed may beachieved independently in various embodiments of the present inventionor may be combined with yet other embodiments, further details of whichcan be seen with reference to the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described embodiments of the invention in general terms,reference will now be made the accompanying drawings, wherein:

FIG. 1 provides a block diagram illustrating a system environment fordelivering advertisements in a mixed reality environment, in accordancewith an embodiment of the invention;

FIG. 2 provides a block diagram illustrating the tailored advertisementdelivery system 200 of FIG. 1 , in accordance with an embodiment of theinvention;

FIG. 3 provides a block diagram illustrating a mixed reality device(s)300 of FIG. 1 , in accordance with an embodiment of the invention;

FIG. 4 provides a block diagram illustrating the computing device system400 of FIG. 1 , in accordance with an embodiment of the invention;

FIG. 5 provides a flowchart illustrating a process flow for deliveringtailored advertisements in a mixed reality environment, in accordancewith an embodiment of the invention;

FIG. 6 provides a flowchart illustrating a process flow for deliveringtailored advertisements in a virtual reality environment, in accordancewith an embodiment of the invention;

FIG. 7 provides a flowchart illustrating a process flow for deliveringtailored advertisements in an augmented reality environment when theuser is using a front faced camera of an augmented reality device, inaccordance with an embodiment of the invention;

FIG. 8 provides a flowchart illustrating a process flow for deliveringtailored advertisements in an augmented reality environment when theuser is using a rear faced camera of an augmented reality device, inaccordance with an embodiment of the invention;

FIG. 9 provides a flowchart illustrating a process flow for analyzingfeatures in a mixed reality environment and identifying one or moreobjects in the mixed reality environment, in accordance with anembodiment of the invention;

FIG. 10 provides a flowchart illustrating a process flow for determininga match between objects in the mixed reality environment of the user andone or more advertisements stored in an advertisement database, inaccordance with an embodiment of the invention;

FIG. 11 provides a flowchart illustrating a process flow for retrievingand displaying advertisements in the mixed reality environment of theuser, in accordance with an embodiment of the invention;

FIG. 12 provides a flowchart illustrating a process flow for determininginteraction between the user and the advertisement presented in themixed reality environment of the user, in accordance with an embodimentof the invention; and

FIG. 13 provides a flowchart illustrating a process flow for collectingand storing interaction metrics associated with the interaction of theuser with the advertisement, in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the present invention will now be described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all, embodiments of the invention are shown. Indeed, theinvention may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. Where possible, any terms expressed in the singularform herein are meant to also include the plural form and vice versa,unless explicitly stated otherwise. Also, as used herein, the term “a”and/or “an” shall mean “one or more,” even though the phrase “one ormore” is also used herein. Furthermore, when it is said herein thatsomething is “based on” something else, it may be based on one or moreother things as well. In other words, unless expressly indicatedotherwise, as used herein “based on” means “based at least in part on”or “based at least partially on.” Like numbers refer to like elementsthroughout.

FIG. 1 provides a block diagram illustrating a system environment fordelivering advertisements in a mixed reality environment, in accordancewith an embodiment of the invention. As illustrated in FIG. 1 , theenvironment 100 includes a tailored advertisement delivery system 200,one or more third party systems 201, an application management system202, user devices of one or more users 110 comprising mixed realitydevice(s) 300 and a computing device system 400.

The tailored advertisement delivery system 200 is a system of thepresent invention for performing one or more process steps describedherein. In some embodiments, the tailored advertisement delivery systemis a cloud computing system. In some embodiments, the tailoredadvertisement delivery system is a non-cloud computing system. The oneor more third party systems 201 may be any systems of a third partyentity (e.g., an advertiser) that bids on one or more objects to promotetheir products or services via advertisements provided by the system ofthe present invention.

The application management system 202 may be any system that provides,maintains, and/or manages one or more applications utilized by the users110, where the applications may be mixed reality applications. In someembodiments, the users 110 may download and store the applicationsprovided by the application management system 202 on the mixed realitydevice(s) 300 and/or the computing device system 400. In someembodiments, the applications provided by the application managementsystem 202 may be virtual reality applications. In some otherembodiments, the applications provided by the application managementsystem 202 may be augmented reality applications. In some embodiments,the application management system 202 may be operated and/or controlledby an application developer who developed the one or more applicationsutilized by the users 110.

The third party entity and an entity that provides the one or moreapplications to the users 110 may be clients of the entity that owns thesystem of the present invention, where the advertisements of productsand/or services provided by the third party entity are displayed in theone or more applications provided by the application management system202. In some embodiments, the advertisements may be displayed in anindependent platform that is not an application provided by theapplication management system 202.

Users 110 may use mixed reality devices(s) 300 and the computing devicesystem 400 to access mixed reality environments. The mixed realitydevice(s) 300 may be any device that allows users to experience a mixedreality environment. Mixed reality environment comprises virtualelements (e.g., computer graphics) and/or real-world elements, where auser can interact with the virtual elements and manipulate the virtualelements. In some embodiments, the mixed reality environment may be avirtual reality environment. An environment that exclusively comprisesvirtual elements with no real-world elements is referred to as a virtualreality environment in the present invention. In some embodiments, themixed reality environment may be an augmented reality environment. Anenvironment where virtual elements overlay on top of the real-worldelements is referred to as an augmented reality environment in thepresent invention.

Mixed reality device(s) 300 may include, but are not limited to,augmented reality headsets, augmented reality lens, augmented realitymirrors, virtual reality headsets, virtual reality lens, or the like.The computer device system 400 may be any device with a camera that iscapable of getting information associated with real-world. In someembodiments, the computing device system 400 may be a mobile phone. Insome embodiments, the computing device system 400 may be a tablet. Insome embodiments, the computing device system 400 is a portable system.In some embodiments, the computing device system 400 is not a portablesystem.

The tailored advertisement delivery system 200, the third party entitysystem 201, the application management system 202, the mixed realitydevice(s) 300 and the computing device system 400 may be in networkcommunication across the system environment 100 through the network 150.The network 150 may include a local area network (LAN), a wide areanetwork (WAN), and/or a global area network (GAN). The network 150 mayprovide for wireline, wireless, or a combination of wireline andwireless communication between devices in the network. In oneembodiment, the network 150 includes the Internet. In general, thetailored advertisement delivery system 200 is configured to communicateinformation or instructions with the third party entity system 201, theapplication management system 202, the mixed reality device(s) 300, andthe computing device system 400 across the network 150.

FIG. 2 provides a block diagram illustrating the tailored advertisementdelivery system 200 in greater detail, in accordance with embodiments ofthe invention. As illustrated in FIG. 2 , in one embodiment of theinvention, the tailored advertisement delivery system 200 includes oneor more processing devices 220 operatively coupled to a networkcommunication interface 210 and a memory device 230.

It should be understood that the memory device 230 may include one ormore databases or other data structures/repositories. The memory device230 also includes computer-executable program code that instructs theprocessing device 220 to operate the network communication interface 210to perform certain communication functions of the tailored advertisementdelivery system 200 described herein. For example, in one embodiment ofthe tailored advertisement delivery system 200, the memory device 230includes, but is not limited to, a network provisioning application 240,neural network models 245, a visual positioning application 250, atailored functionality generation application 255, a bidding application260, a metric collection and an analysis application 265, and a datarepository 290 comprising an advertisement database 292, a metricsdatabase 294, user profiles 296, and any other data received, processed,or accessed by one or more applications in the memory device 230. Thecomputer-executable program code of the network provisioning application240, the neural network models 245, the visual positioning application250, the tailored functionality generation application 255, the biddingapplication 260, and the metric collection and analysis application 265may instruct the processing device 220 to perform certain logic,data-processing, and data-storing functions of the tailoredadvertisement delivery system 200 described herein, as well ascommunication functions of the tailored advertisement delivery system200.

The network provisioning application 240, the neural network models 245,the visual positioning application 250, the tailored functionalitygeneration application 255, the bidding application 260, and the metriccollection and analysis application 265 are configured to invoke or usethe data in the data repository 290 when communicating through thenetwork communication interface 210 with the tailored advertisementdelivery system 200, and/or the computing device system 400. In someembodiments, the network provisioning application 240, the neuralnetwork models 245, the visual positioning application 250, the tailoredfunctionality generation application 255, the bidding application 260,and/or the metric collection and analysis application 265 may store thedata extracted or received from the mixed reality devices 300 and thecomputing device system 400 in the data repository 290.

In some embodiments, the network provisioning application 240, theneural network models 245, the visual positioning application 250, thetailored functionality generation application 255, the biddingapplication 260, and the metric collection and analysis application 265may be a part of a single application of the tailored advertisementdelivery system 200 that perform steps of the process flows describedherein. The neural network models 245 may identify one or more featuresor objects in a real-world of the user 110 and may also identifyreal-time emotions of the user 110. The tailored advertisement deliverysystem 200 trains the neural network models 245 to identify real-worldobjects and features and also to identify real-time time emotions of theuser 110. The visual positioning application 250 may create a spatialcloud of the user 110 that comprises one or more objects in anenvironment of the user 110 and coordinates of the one or more objectswith respect to the position of the user 110 or position of the cameraof the user device. The tailored functionality generation application255 may generate tailored advertisements to display to the user 110. Thetailored functionality generation application 255 may also calculatepositional coordinates to place the generated advertisement in a mixedreality environment of the user 110. The bidding application 260 is aplatform provided by the tailored advertisement delivery system 200 thatcomprises a user interface which allows the third party entity systemsto bid on one or objects (e.g., water bottle, or the like). The metriccollection and analysis application 265 may collect metrics associatedwith interactions of the user 110 with the advertisements displayed tothe user 110 in a mixed reality environment. The metric collection andanalysis application may also analyze the metrics to determine likes anddislikes of the user 110. The network provisioning application 240, theneural network models 245, the visual positioning application 250, thetailored functionality generation application 255, the biddingapplication 260, and the metric collection and analysis application 265may perform additional steps of the process flow as described in FIG. 5through FIG. 13 .

FIG. 3 provides a block diagram illustrating a mixed reality device(s)300 of FIG. 1 , in accordance with an embodiment of the invention. Asillustrated in FIG. 3 , in one embodiment of the invention, the mixedreality device 300 includes a processing unit 310 operatively coupledwith network communication unit 315, a memory 340, a control unit 320,an input unit 305, an optical unit 325, and an output unit 330. Thecontrol unit 320 may control the input unit 305, the optical unit 325,and/or

the output unit 330 of the mixed reality device 300 based on theinstructions received from the processing unit 310. The input unit 305may be a component that captures the real-world objects in anenvironment of the user. In an exemplary embodiment, the input unit 305may be lens. The optical unit 325 may be a component that allows theuser to experience a mixed reality environment. In an exemplaryembodiment, the optical unit 325 may be a headset. In some embodiments,the optical unit 325 and the input unit 305 may be a single unit. Insuch embodiments, the single unit is capable of operating as anindependent unit that is in constant communication with other componentsof the mixed reality device 300. The memory unit 340 comprises atailored advertisement delivery application 342 which is provided by thetailored advertisement delivery system 200. The tailored advertisementdelivery application 342 may provide one or more instructions to theprocessing unit 310 which in turn causes the control unit 320 to controlthe input unit 305, the optical unit 325, and/or the output unit 330 toperform one or more steps as described herein. In an exemplaryembodiment, the machine learning models 423 present in the tailoredadvertisement delivery application 342 may cause the input unit 305 tocapture the real-time objects in the environment of the user 110 andidentify the type of real-time objects. The memory 340 may furthercomprise one or more applications (e.g., virtual reality application,augmented reality applications, or the like) provided by the applicationmanagement system 202. For example, the user 110 may experience mixedreality environment via the one or more applications provided by theapplication management system 202 and one or more components of themixed reality device 330.

In some embodiments, based on the instructions and control signalsreceived from the tailored advertisement delivery system 200 or theexecutable instructions present in the tailored advertisement deliveryapplication 342, the processing unit 310 via the control unit 320 maycause the mixed reality device 300 to perform one or more actionsdescribed herein.

FIG. 4 provides a block diagram illustrating a computing device system400 of FIG. 1 in more detail, in accordance with embodiments of theinvention. However, it should be understood that a mobile telephone ismerely illustrative of one type of computing device system 400 that maybenefit from, employ, or otherwise be involved with embodiments of thepresent invention and, therefore, should not be taken to limit the scopeof embodiments of the present invention.

Some embodiments of the computing device system 400 include a processor410 communicably coupled to such devices as a memory 420, user outputdevices 436, user input devices 440, a network interface 460, a powersource 415, a clock or other timer 450, a camera 480, and a positioningsystem device 475. The processor 410, and other processors describedherein, generally include circuitry for implementing communicationand/or logic functions of the computing device system 400. For example,the processor 410 may include a digital signal processor device, amicroprocessor device, and various analog to digital converters, digitalto analog converters, and/or other support circuits. Control and signalprocessing functions of the computing device system 400 are allocatedbetween these devices according to their respective capabilities. Theprocessor 410 thus may also include the functionality to encode andinterleave messages and data prior to modulation and transmission. Theprocessor 410 can additionally include an internal data modem. Further,the processor 410 may include functionality to operate one or moresoftware programs, which may be stored in the memory 420. For example,the processor 410 may be capable of operating a connectivity program,such as a web browser application 422. The web browser application 422may then allow the computing device system 400 to transmit and receiveweb content, such as, for example, location-based content and/or otherweb page content, according to a Wireless Application Protocol (WAP),Hypertext Transfer Protocol (HTTP), and/or the like.

The processor 410 is configured to use the network interface 460 tocommunicate with one or more other devices on the network 150. In thisregard, the network interface 460 includes an antenna 476 operativelycoupled to a transmitter 474 and a receiver 472 (together a“transceiver”). The processor 410 is configured to provide signals toand receive signals from the transmitter 474 and receiver 472,respectively. The signals may include signaling information inaccordance with the air interface standard of the applicable cellularsystem of the wireless network 152. In this regard, the computing devicesystem 400 may be configured to operate with one or more air interfacestandards, communication protocols, modulation types, and access types.By way of illustration, the computing device system 400 may beconfigured to operate in accordance with any of a number of first,second, third, and/or fourth-generation communication protocols and/orthe like.

As described above, the computing device system 400 has a user interfacethat is, like other user interfaces described herein, made up of useroutput devices 436 and/or user input devices 440. The user outputdevices 436 include a display 430 (e.g., a liquid crystal display or thelike) and a speaker 432 or other audio device, which are operativelycoupled to the processor 410.

The user input devices 440, which allow the computing device system 400to receive data from a user such as the user 110, may include any of anumber of devices allowing the computing device system 400 to receivedata from the user 110, such as a keypad, keyboard, touch-screen,touchpad, microphone, mouse, joystick, other pointer device, button,soft key, and/or other input device(s). The user interface may alsoinclude a camera 480, such as a digital camera.

The computing device system 400 may also include a positioning systemdevice 475 that is configured to be used by a positioning system todetermine a location of the computing device system 400. For example,the positioning system device 475 may include a GPS transceiver. In someembodiments, the positioning system device 475 is at least partiallymade up of the antenna 476, transmitter 474, and receiver 472 describedabove. For example, in one embodiment, triangulation of cellular signalsmay be used to identify the approximate or exact geographical locationof the computing device system 400. In other embodiments, thepositioning system device 475 includes a proximity sensor ortransmitter, such as an RFID tag, that can sense or be sensed by devicesknown to be located proximate a merchant or other location to determinethat the computing device system 400 is located proximate these knowndevices.

The computing device system 400 further includes a power source 415,such as a battery, for powering various circuits and other devices thatare used to operate the computing device system 400. Embodiments of thecomputing device system 400 may also include a clock or other timer 450configured to determine and, in some cases, communicate actual orrelative time to the processor 410 or one or more other devices.

The computing device system 400 also includes a memory 420 operativelycoupled to the processor 410. As used herein, memory includes anycomputer readable medium (as defined herein below) configured to storedata, code, or other information. The memory 420 may include volatilememory, such as volatile Random Access Memory (RAM) including a cachearea for the temporary storage of data. The memory 420 may also includenon-volatile memory, which can be embedded and/or may be removable. Thenon-volatile memory can additionally or alternatively include anelectrically erasable programmable read-only memory (EEPROM), flashmemory or the like.

The memory 420 can store any of a number of applications which comprisecomputer-executable instructions/code executed by the processor 410 toimplement the functions of the computing device system 400 and/or one ormore of the process/method steps described herein. For example, thememory 420 may include such applications as a conventional web browserapplication 422, a tailored advertisement delivery application 421comprising Machine Learning (ML) models 423, an SMS application, or thelike. The memory 420 may further comprise one or more applications(e.g., virtual reality application, augmented reality applications, orthe like) provided by the application management system 202. The memory420 of the computing device system 400 may comprise a Short MessageService (SMS) application 424 configured to send, receive, and storedata, information, communications, alerts, and the like via the wirelesstelephone network 152. In some embodiments, the tailored advertisementdelivery application 421 provided by the tailored advertisement deliverysystem 200 causes the one or more components of the computing devicesystem 400 to capture the real-time objects in the environment of theuser 110 and identify the type of real-time objects.

The memory 420 can also store any of a number of pieces of information,and data, used by the computing device system 400 and the applicationsand devices that make up the computing device system 400 or are incommunication with the computing device system 400 to implement thefunctions of the computing device system 400 and/or the other systemsdescribed herein.

FIG. 5 provides a flowchart illustrating a high level process flow 500for delivering tailored advertisements in a mixed reality environment,in accordance with an embodiment of the invention. Mixed realityenvironment is a combination of virtual reality environment andaugmented reality environment, where virtual elements (e.g.,advertisements, gaming characters in one or more games, or the like) areplaced around real-world objects that are in the environment of theuser. The user may interact and modify the virtual elements displayedusing one or more components of the mixed reality devices 300 orcomputing device systems 400. In some embodiments, the user may interactand modify the virtual elements using gestures, expressions, or thelike. In such embodiments, the system of the present invention supportsthe user's interactions by transmitting one or more instructions to themixed reality devices 300 and/or the computing device system 400.

As shown in block 505, the system continuously detects features in amixed reality environment. In some embodiments, the tailoredadvertisement delivery application 342 and the tailored advertisementdelivery application 421 provided by the tailored advertisement deliverysystem 200 cause the one or more components of the mixed reality devices300 and the computing device system 400 to detect features in a mixedreality environment. For example, the tailored advertisement deliveryapplication may cause a mixed reality headset to detect and capture theenvironment of the user 110. In some embodiments, the tailoredadvertisement delivery application 342 and the tailored advertisementdelivery application 421 may comprise executable instructions that arestored in the memory of the mixed reality device 300 and the computingdevice systems 400.

As shown in block 510, the system identifies one or more objects in themixed reality environment. In some embodiments, the tailoredadvertisement delivery application provided by the tailoredadvertisement delivery system 200 comprises machine learning models thatare trained to identify real-world objects. The tailored advertisementdelivery system 200 trains neural networks using several images andgenerates the machine learning models that can identify real-worldobjects, where the machine learning models are then transmitted andstored on the user device. In some embodiments, the machine learningmodels are trained to identify objects that are associated with productsmanufactured, distributed, and/or supplied by the third party entities(e.g., advertisers). For example, the system may identify that a thirdparty entity manufactures facial tissues and is bidding on any tissuepapers or any facial products and may train the neural networks overseveral images of tissue papers and facial products. After completion ofthe training, the tailored advertisement delivery system 200 developsmachine learning models that can identify any facial products and tissuepapers with a confidence score attached to each object.

In alternate embodiments, the tailored advertisement deliveryapplication provided by the tailored advertisement delivery system 200transmits the real-time data captured by the user device to the tailoredadvertisement delivery system 200, where the identification of the oneor more objects that are captured by the one or more components of theuser devices takes place in the tailored advertisement delivery system200. The process of analyzing features and identifying objects isexplained in more detail in FIG. 9 .

As shown in block 515, the system identifies that the one or moreobjects match at least one of the advertising targets. The tailoredadvertisement delivery system 200 comprises the advertisement database292 that comprises advertisement context provided by the one or morethird party systems via the bidding application 260. For example, a softdrink manufacturing company may bid on a water bottle and upon when awater bottle is detected in the environment of the user in real-time,the system displays an advertisement for a soft drink manufactured bythe soft drink company. As shown in block 520, the system generates atailored advertisement and transmits the tailored advertisement to theuser device of the user. The tailored advertisement delivery system 200generates a tailored advertisement that can be displayed to the userwithout any interruptions and can possibly be converted into a “hit”(e.g., user clicking/interacting with an advertisement) and thentransmits the tailored advertisement to the user device. The process ofselecting and generating the tailored advertisement is dependent onmultiple factors and is explained in greater detail in FIG. 10 and FIG.11 .

As shown in block 525, the system causes the user device to display thetailored advertisement. The tailored advertisement delivery system 200causes the one or more components of the user device to display thetailored advertisements to the user, where the tailored advertisementblends into the environment of the user. In an exemplary embodiment,where the mixed reality environment is a virtual reality environment andthe user is moving in the virtual world, the tailored advertisementdelivery system 200 may display an advertisement comprising an offerrelated to a brand of car outside a virtual car showroom. The process ofdisplaying the tailored advertisement in the environment of the user isexplained in greater detail in FIG. 11 .

As shown in block 530, the system determines that the user isinteracting with the tailored advertisement in real-time. Continuingwith the previous example, the tailored advertisement delivery systemafter displaying the advertisement to the user may continuously monitoractions of the user and may identify that the user has clicked on theoffer. The process of determining that the user interacting with thetailored advertisement is explained in greater detail in FIG. 12 .

As shown in block 535, the system collects and stores one or moremetrics associated with the interaction. The tailored advertisementdelivery system 200 based on continuously monitoring the actions of theusers, collects one or more metrics associated with the interaction ofthe user with the tailored advertisement. The process of collecting andstoring the one or more metrics is explained in greater detail in FIG.13 .

The process flow 500 described in FIG. 5 may vary based on whether themixed reality environment is a virtual reality environment or anaugmented reality environment. In an embodiment, where the mixed realityenvironment is a virtual reality environment, the process flow fordelivering tailored advertisements is explained in FIG. 6 . In anembodiment where the mixed reality environment is an augmented realityenvironment and where the augmented reality environment is experiencedby the user using a front view camera of a user device (e.g., frontfacing camera of a mobile device), the process flow for deliveringtailored advertisements is explained in FIG. 7 . The process flowdescribed in FIG. 7 is also applicable for any mixed reality devices(e.g., augmented reality mirrors or the like) that allow the user toview bodily features (e.g., face) and the environment around the user.In an embodiment where the mixed reality environment is an augmentedreality environment and where the augmented reality environment isexperienced by the user using a rear view camera of a user device (e.g.,rear view camera of a mobile device), the process flow for deliveringtailored advertisements is explained in FIG. 8 . The process flowdescribed in FIG. 8 is also applicable for other mixed reality devicessuch as augmented reality headsets, augmented reality lenses, or thelike that allow the user to view the real-time environment via anoptical unit.

FIG. 6 provides a flowchart illustrating a process flow 600 fordelivering tailored advertisements in a virtual reality environment, inaccordance with an embodiment of the invention. As shown in block 605,the system identifies that the user has entered a virtual realityenvironment of an application. The user may access the virtual realityenvironment via one or more applications provided by the applicationmanagement systems 202. Virtual reality environment provided by eachapplication may vary. For example, a virtual reality environmentprovided by a gaming application may vary from a virtual realityenvironment provided by an application that is designed to view anaquarium. The one or more applications may be stored on a user device(e.g., virtual reality glasses or a console linked with the virtualreality glasses) and the user may access the one or more applications byproviding authentication details. The tailored advertisement deliverysystem 200 may track the one or more applications that are installed onthe user device via the tailored advertisement delivery application todetermine that the user has accessed a virtual reality application andentered the virtual environment provided by the application.

In some embodiments, the tailored advertisement delivery system 200 upondetermining that the user has accessed the virtual reality application,may generate a unique identifier, where the unique identifier is used totrack and store information associated with the user interactions withthe virtual reality application.

As shown in block 610, the system identifies one or more virtual objectsaround the user in the virtual environment. For example, where thevirtual reality application is an application that allows user toexperience an aquarium, the tailored advertisement delivery system 200may identify that the user is viewing a jelly fish tank. The tailoredadvertisement delivery system 200 identifies the one or more objects theuser is viewing via the one or more components of the user device andthe tailored advertisement delivery application stored on the userdevice. In some embodiments, the tailored advertisement delivery system200 may have information (e.g., features provided by the virtual realityapplication, virtual objects and positions of the virtual objects withinthe virtual environment provided by the virtual reality application, orthe like) associated with the one or more applications stored in thedata repository, where this information is received from the applicationmanagement system 202.

As shown in block 615, the system determines current location of theuser based on the one or more objects in the virtual environment. Basedon the information associated with the virtual reality application andthe one or more objects identified in block 610, the tailoredadvertisement delivery system 200 tracks the position of the user withinthe virtual reality environment. In response to identifying the one ormore objects and the current location of the user, the system maygenerate tailored advertisements associated with the one or moreobjects, virtual stores around the current location of the user, and thelike. The process flow associated with generation of tailoredadvertisements in discussed in detail in FIG. 10 and FIG. 11 . In someembodiments, the generated tailored advertisement may be a twodimensional (2D) advertisement. In some other embodiments, the generatedtailored advertisement may be a three dimensional (3D) advertisement. Insome embodiments, the generated tailored advertisement may be a videoadvertisement.

As shown in block 620, the system calculates positional coordinates toplace an advertisement based on the current location of the user. Thetailored advertisement delivery system 200 calculates the positionalcoordinates based on the type of tailored advertisement. For example, ifthe tailored advertisement is a 2D advertisement, the system determinesone or more billboards around the current location of the user andcalculates the positional coordinates to place the 2D ad. In anotherexample, if the generated tailored advertisement is a 3D advertisement,the system determines a place on the ground around the user (e.g.,preferably within a 0.01 mile radius of the user) where there are novirtual objects and calculates the positional coordinates of the placeto display the three dimensional advertisement.

As shown in block 625, the system, in some embodiments, automaticallyplaces the advertisement in the positional coordinates. In someembodiments, the tailored advertisement delivery system 200 may havepermissions to automatically place the tailored advertisement in thevirtual reality environment and in such embodiments, the systemautomatically places the advertisement in the calculated positionalcoordinates. As shown in block 630, the system, in alternateembodiments, transmits the positional coordinates and the tailoredadvertisement to the application management system of the virtualreality application. The application management system upon receivingthe positional coordinates places the tailored advertisement in thespecified positional coordinates.

FIG. 7 provides a flowchart illustrating a process flow 700 fordelivering tailored advertisements in an augmented reality environmentwhen the user is using a front faced camera of an augmented realitydevice, in accordance with an embodiment of the invention.

As shown in block 705, the system determines that the user is using anaugmented reality device. In some embodiments, the tailoredadvertisement delivery system 200 may receive an indication from theaugmented reality device once the user accesses the augmented realitydevice. As shown in block 710, the system determines that the user isaccessing an augmented reality application via the augmented realitydevice. The user may experience an augmented reality environment basedon accessing one or more augmented reality applications provided by theapplication management system 202. The virtual elements provided by eachaugmented reality application may vary. For example, virtual elementsprovided by a gaming application may vary from virtual elements providedby a shopping application. The one or more augmented realityapplications may be stored on a user device (e.g., computing devicesystem, augmented reality glasses, or a console linked with theaugmented reality glasses) and the user may access the one or moreaugmented reality applications by providing authentication details. Thetailored advertisement delivery system 200 may track the one or moreapplications that are installed on the user device via the tailoredadvertisement delivery application to determine that the user hasaccessed an augmented reality application.

As shown in block 715, the system determines that the user is using afront facing camera of the augmented reality device to access featuresof the augmented reality application. In some embodiments, the frontfaced camera is typically used by augmented reality applications thatare associated with providing shopping experience (e.g., to try outclothes, makeup, or the like) to the users. In some embodiments, thesystem may determine that the user is using the front facing camerabased on gathering information from the one or more components of theuser device. In other embodiments, the system may determine that theuser is using the front facing camera based on determining that the userhas accessed an augmented reality application that requires usage of thefront facing camera of the user device.

As shown in block 720, the system identifies, via neural network models,one or more bodily features of the user. In some embodiments, thesystem, via neural network models, identifies only one or more facialfeatures (e.g., eyes, nose, mouth, or the like) of the user. In someother embodiments, the system identifies all of the one or more bodilyfeatures (e.g., feet, hands, or the like) of the user. In response todetermining that the user is using the front facing camera of the userdevice, the tailored advertisement delivery system 200 identifies theone or more bodily features of the user, via the ML models in thetailored advertisement delivery application present in the user device.In some other embodiments, the tailored advertisement delivery system200 receives the real-time feed from the front faced camera of the userdevice, via the tailored advertisement delivery application and one ormore components of the user device, and then identifies the one or morebodily features via the trained neural network models/machine learningmodels.

In some embodiments, identification of the one or more bodily featurescomprises the neural network models generating a mesh that overlaps withthe one or more bodily features of the user. The mesh generated by theneural network models may be divided into one or more segments. Theneural networks may determine a type of the bodily features based on thegenerated mesh. For example, the system may detect a feature and maygenerate a mesh that overlaps the detected feature. Based on the mesh,the neural network models may determine that the feature detected isnose of the user. In another example, the system may detect user's feetand may generate a mesh that overlaps the user's feet.

As shown in block 725, the system transmits an advertisement to anapplication management system associated with the augmented realityapplication, where the advertisement overlaps at least one of the one ormore bodily features. The advertisement generated by the tailoredadvertisement delivery system 200 is based on the one or more bodilyfeatures identified in block 720. For example, the system identifiesface of the user and presents advertisements associated with make-up tothe user. In another example, the system identifies entire body andpresents advertisements associated with clothing to the user. Theadvertisement is generated by the tailored advertisement delivery system200 based on one or more factors (e.g., user preferences, historicaldata, user profile, or the like). The one or more factors that affectthe generation of the tailored advertisement is explained in detail inFIG. 10 and FIG. 11 . In some embodiments, the system transmits theadvertisement to the application management system of the augmentedreality application, where the application management system displaysthe advertisement to the user. In some embodiments, the system alsotransmits the coordinates of the one or more segments to place theadvertisement, where the one or more segments are associated with themesh that is generated by the neural network models. In alternateembodiments, as shown in block 730, the system automatically displays anadvertisement, where the advertisement overlaps at least one of the oneor more bodily features. The system may calculate coordinates of the oneor more segments to place the advertisement and places the advertisementin the coordinates. For example, the system may determine thatcoordinates X, Y, and Z associated with one or more segments of the meshassociated with user's eyes are suitable to place a shade of an eyeshadow such that the eye shadow shade overlaps with the one or moresegments of the user's eyelids. In some embodiments, based on thecoordinates, the system may automatically alter the generatedadvertisement such that the advertisement overlaps the one or morebodily features.

In some embodiments, the advertisement may be an interstitialadvertisement, where the advertisement completely covers the environmentof the user. In some such embodiments, calculation of the coordinatesmay not be required. For example, if a user is trying out clothes usingthe front facing camera of a mobile phone, where the user is viewingdifferent outfits via the user interface of the mobile device, thesystem may completely transform the user interface to display theadvertisement. In other embodiments, the advertisement is not aninterstitial advertisement.

FIG. 8 provides a flowchart illustrating a process flow 800 fordelivering tailored advertisements in an augmented reality environmentwhen the user is using a rear faced camera of an augmented realitydevice, in accordance with an embodiment of the invention.

As shown in block 805, the system determines that the user is using anaugmented reality device. In some embodiments, the tailoredadvertisement delivery system 200 may receive an indication from theaugmented reality device once the user accesses the augmented realitydevice. As shown in block 810, the system determines that the user isaccessing an augmented reality application via the augmented realitydevice.

As shown in block 815, the system determines that the user is using arear facing camera of the augmented reality device to access features ofthe augmented reality application. In some embodiments, the augmentedreality application may be an application specifically designed toutilize rear facing camera of the user device. In such an embodiment,the tailored advertisement delivery system 200 may identify that theuser is using the rear facing camera based on identifying that the userhas accessed the augmented reality application. In some embodiments, thetailored advertisement delivery system 200 may identify that the user isusing the rear facing camera based on communicating with the one or morecomponents of the user device.

As shown in block 820, the system continuously identifies, via neuralnetwork models, one or more objects in the environment of the user. Inresponse to determining that the user is using the rear facing camera ofthe user device, the tailored advertisement delivery system 200continuously captures the environment of the user and identifies the oneor more objects in the environment of the user, via the ML models in thetailored advertisement delivery application present in the user device.In some other embodiments, the tailored advertisement delivery system200 receives the real-time feed from the rear faced camera of the userdevice, via the tailored advertisement delivery application and one ormore components of the user device, and then identifies the one or moreobjects via the trained neural network models/machine learning models.

As shown in block 825, the system determines the positional coordinatesof the one or more objects relative to the camera of the user device.For example, if a water bottle is identified in the environment of theuser, the system calculates the positional coordinates of the waterbottle with respect to the camera. In some other embodiments, the systemmay determine the positional coordinates of the one or more objectsrelative to any component of the user device. In some embodiments, thetailored advertisement delivery system 200 may generate output boundingboxes for each object and use a pixel detection algorithm to calculatethe distance from each object.

As shown in block 830, the system builds a spatial cloud of theenvironment of the user. The spatial cloud comprises informationassociated with the one or more objects (e.g., type of the object, orthe like) and positional coordinates of the one or more objects. In someembodiments, the system may store information and positional coordinatesassociated with some objects of the one or more objects identified inthe environment of the user, where those objects are associated with theobjects that are associated with one or more advertisements stored inthe database. For example, if the system identifies a water bottle, alaptop, a coffee mug, and a plant, the system may identify that thethird party entities placed their bids on water bottle and on laptop andmay store information associated with the water bottle and the laptop.In some embodiments, the system may store information associated withall objects identified in the environment of the user. Continuing withthe previous example, the system may store information associated withwater bottle, laptop, coffee mug, and plant.

As shown in block 835, the system calculates one or more positionalcoordinates to place an advertisement based on the spatial cloud. Thecalculation of the one or more positional coordinates may be based onthe type of advertisement generated by the system. In an embodiment,where the generated advertisement is a 2D advertisement, the tailoredadvertisement delivery system 200 may identify plain surface to displaythe 2D advertisement. In such embodiments, where the generatedadvertisement is a 2D advertisement that is placed in a mixed realityenvironment which is a three dimensional environment, the systemconverts the 2D advertisement into frames that fits into the threedimensional mixed reality environment.

In another embodiment, where the generated advertisement is a 3Dadvertisement, the tailored advertisement delivery system 200 may placethe 3D on the ground where there are no other objects. In someembodiments, the one or more positional coordinates are calculated suchthat the advertisement is within a predefined distance (e.g., 0.01 mileradius from the camera o the user devices) from the current location ofthe user. In some embodiments, the tailored advertisement deliverysystem 200 may calculate the one or more positional coordinates suchthat the advertisement is placed close to the object that triggered thegeneration of the advertisement. For example, the system may place anadvertisement associated with a soft drink close to a water bottle,where the water bottle triggered the generation of the soft drinkadvertisement.

As shown in block 840, the system transmits the one or more positionalcoordinates to place the advertisement to an application managementsystem associated with the augmented reality application. Theapplication management system may then place the advertisement in any ofthe one or more positional coordinates provided by the tailoredadvertisement delivery system 200. In alternate embodiments, as shown inblock 845, the system automatically displays the advertisement in the atleast one of the one or more positional coordinates.

In some embodiments, the advertisement may be an interstitialadvertisement, where the advertisement completely covers the environmentof the user. In some such embodiments, calculation of the one or morepositional coordinates may not be required. For example, if a user isplaying a game using the rear facing camera of a mobile phone, where theuser is viewing the game via the user interface of the mobile device,the system may completely transform the user interface to display theadvertisement. In other embodiments, the advertisement is not aninterstitial advertisement.

FIG. 9 provides a flowchart illustrating a process flow 900 foranalyzing features in a mixed reality environment and identifying one ormore objects in the mixed reality environment, in accordance with anembodiment of the invention.

As shown in block 905, the system continuously captures environment ofthe user. As explained above, the tailored advertisement delivery system200 captures the environment of the user via the tailored advertisementdelivery application and the one or more components of the user device(e.g., computing device system 400 or mixed reality device 300).

As shown in block 910, the system analyzes the environment of the userto identify one or more objects. As explained above, neural networkmodels are trained using images and machine learning models aregenerated by the tailored advertisement delivery system 200, where themachine learning models identify the one or more objects with aconfidence score associated with the identification. For example, themachine learning model may detect an object around the user and mayidentify that the object is a water bottle with 99% confidence score. Insome embodiments, the tailored advertisement delivery system 200 may usetraditional vision techniques for detecting and identifying the one ormore objects. In some embodiments, the tailored advertisement deliverysystem 200 may use object detection networks like mobile-net foridentifying the one or more objects.

As shown in block 915, the system determines type of the one or moreobjects. The type of the object may disclose information whether anobject is an outdoor object, indoor object, or the like. For example,the system may identify sunglasses and may classify sunglasses as anoutdoor object. The tailored advertisement delivery system 200 may trainneural networks and generate machine learning models that identifywhether the objects are indoor objects, outdoor objects, objects relatedto a specific location (e.g., gym, office, or the like), or the like.

As shown in block 920, the system determines a location of the userbased on analyzing the one or more objects. The tailored advertisementdelivery system 200 determines the location of the user based onidentifying the type of the one or more objects. For example, the systemmay identify telephone, laptop, books, notepads, stapler, or the like inthe environment of the user and may determine that the user is in anoffice. In another example, the system may identify treadmill, weights,training bench, or the like in the environment of the user and maydetermine that the user is in a gym. In another example, the system mayidentify knives, vegetables, pots, or the like in the environment of theuser and the system may identify that the user is in a kitchen. In someembodiments, in addition to identifying the location of the user basedon analyzing the one or more objects, the tailored advertisementdelivery system 200 may also use at least one of global positioningsystem, Bluetooth, WLAN, to determine the location of the user.

In some embodiments, the tailored advertisement delivery system 200determines time of the day based on analyzing the environment of theuser. In some embodiments, the system may also identify a brandassociated with the one or more object. For example, the system mayidentify a water bottle in the environment of the user and may determinethe brand of the water bottle using Optical Character Recognition (OCR).In some embodiments, the tailored advertisement delivery system 200 mayidentify that at least one object appears more than a predeterminedamount of times in the environment of the user and may store informationassociated with the at least one object in a profile of the user. Insome embodiments, the tailored advertisement delivery system 200 mayidentify one or more patterns associated with the user, where the one ormore patterns may be associated with activities of the user, location ofthe user, or the like. As shown in block 930, the system stores therendered data in a profile of the user.

FIG. 10 provides a flowchart illustrating a process flow 1000 fordetermining a match between objects in the mixed reality environment ofthe user and one or more advertisements stored in an advertisementdatabase, in accordance with an embodiment of the invention.

As shown in block 1005, the system identifies a first object in theenvironment of the user. As shown in block 1010, the system determines amatch for the first object in an advertisement database. The system maydetermine a category associated with the first object and may determineone or more advertisements in the advertisement for the categoryassociated with the first object. For example, the system may identify awater bottle in the environment of the user, may determine that waterbottle is associated with consumable drinks, and may search theadvertisement database to identify one or more advertisements associatedwith the category of consumable drinks and to determine a match for thewater bottle. The match is associated with a bid placed by a third partyentity that manufactures, distributes, and/or sells consumable drinks.In some cases, the bid placed by the third party entity may be a winningbid.

As shown in block 1015, the system determines that more than oneadvertisements exist in the advertisement database for the first object.Continuing with the previous example, the third party entity that placeda bid on the water bottle may have one or more advertisement contextsthat are directed to different products targeted to different types ofusers. The third party entity may have one or more campaigns that targetdifferent types of users. In one exemplary embodiment, one campaign maybe associated with users that are conscious about health. In anotherexemplary embodiment, one campaign may be associated with a younger agegroup.

As shown in block 1020, the system selects a first advertisement fromthe more than advertisements based on one or more factors. The one ormore factors may include, but are not limited to, location of the user,time of the day, age group of the user, patterns associated withactivities of the user, user preferences, or the like. In someembodiments, the information associated with the one or more factors maybe stored in a profile of the user. Continuing with the previousexample, the system may identify that the user falls under a 12-17 agegroup and may present an advertisement associated with an orangeflavored soft drink to the user. In another case, the system mayidentify that the user is health conscious based on identifying that theuser goes to a gym every day and may present an advertisement associatedwith zero calorie soft drink to the user. In another case, the systemmay identify that the user is in a playground and may present anadvertisement of an energy drink to the user. In another case, thesystem may present an advertisement associated with a caffeinated drinkto the user in the morning. In some embodiments, the tailoredadvertisement delivery system 200 may select the first advertisementbased on any combination of the one or more factors mentioned above.

In some embodiments, the tailored advertisement delivery system 200 maydisplay an advertisement associated with a third party entity thatplaced the highest bid upon identification of the object associated withthe bid. In some embodiments, the tailored advertisement delivery system200 may assign higher priority to the advertisements associated with athird party entity that placed the second highest bid based on thepreferences of the user. In some embodiments, the system may recycleadvertisements such that the advertisements that are displayed to theuser are non-repetitive. For example, the system may identify a secondobject associated with the same category as that of the first object andthe system may display any other advertisement other than the firstadvertisement based on the one or more factors.

FIG. 11 provides a flowchart illustrating a process flow 1100 forretrieving and displaying advertisements in the mixed realityenvironment of the user, in accordance with an embodiment of theinvention.

As shown in block 1105, the system retrieves context associated with thefirst advertisement. In response to selecting the first advertisementfrom the more than one advertisements, the tailored advertisementdelivery system 200 retrieves context associated with the firstadvertisement. Continuing with the previous example, the system mayselect an advertisement associated with zero calorie soft drink and mayretrieve context associated with the zero calorie drink. Context mayinclude, but is not limited to, tag lines, available advertisementformats (e.g., 2D, 3D, or the like), audio, video, images, or the likeassociated with the advertisement.

As shown in block 1110, the system generates visual content comprisingthe context associated with the first advertisement based on a set offactors. The set of factors may include, but are not limited to, type ofmixed reality environment (e.g., augmented reality environment orvirtual reality environment), type of application used to access themixed reality environment (gaming augmented reality application, anaquarium virtual reality application, or the like), the spatial cloud,the one or more objects in the environment of the user, time availablefor displaying the advertisement, space available to display theadvertisement, or the like. In some embodiments, the tailoredadvertisement delivery system 200 may use any combination of the set offactors to generate the visual content associated with the firstadvertisement. The generated visual content may comprise a 2Dadvertisement or a 3D advertisement. In the case where the generatedadvertisement is a 3D advertisement, the 3D advertisement may compriseat least one of a 3D animation, a 3D image, and/or a 3D video. In thecase where the generated advertisement is a 2D advertisement, the 2Dadvertisement may comprise at least one of a 2D image and/or a 2D video.

In an exemplary embodiment, the system may determine that the type ofapplication used by the user to access the mixed reality environment isa gaming application with animated characters and may generate 3Danimated visual content comprising the context of the firstadvertisement. In another exemplary embodiment, the system may determinethat the type of application used by the user to access the mixedreality environment is a virtual shopping application that allows usersto visit a virtual shopping mall. In one case, the system may generate avisual 2D advertisement that fits a virtual bill board located within apredefined distance from the current location of the user. In anothercase, the system may generate a video advertisement that fits a virtualbill board located within a predefined distance from the currentlocation of the user. In some embodiments, the system may determine thatthe length of the video advertisement exceeds the time available todisplay the advertisement and may instead display a visual 3Dadvertisement. The tailored advertisement delivery system 200 maydetermine the time available to display the advertisement based at leaston user patterns, type of application, likes and dislikes of the user,or the like. In some embodiments, the system may determine that thespace around the user is crowded and may generate a 3D advertisementthat can be placed on the ground. The visual content generated by thetailored advertisement delivery system 200 is enhanced so thatadvertisement blends into the mixed reality environment of the user. Insome embodiments, the tailored advertisement delivery system 200 may useGenerative Adversarial Networks (GAN) to enhance the visual content. Insome embodiments, the visual content generated by the system may be aninterstitial advertisement, where the visual content completelytransforms the environment of the user to display the context of theadvertisement.

As shown in block 1115, the system compresses the visual content togenerate a compressed package, where the compressed package furthercomprises positional coordinates to place the advertisement andinteraction gestures to interact with the visual content. In someembodiments, where the mixed reality environment is an augmented realityenvironment, the interaction gestures comprise at least one of handgestures, eye gestures, facial expressions, or the like. For example,the user may interact with the visual content by gazing at theadvertisement for more than the predetermined amount of time. In someembodiments, where the mixed reality environment is a virtual realityenvironment, the user may interact with the visual content via acontroller and one or more components of the virtual reality device.

As shown in block 1120, the system transmits the compressed package tothe user device. As shown in block 1125, the system causes the userdevice to decompress the compressed package. As shown in block 1130, thesystem displays the visual content in the environment of the user. Insome embodiments, the system transmits the compressed package to theapplication management system of the first application.

FIG. 12 provides a flowchart illustrating a process flow 1200 fordetermining interaction between the user and the advertisement presentedin the mixed reality environment of the user, in accordance with anembodiment of the invention.

As shown in block 1205, the system in response to displaying the visualcontent, determine gaze of the user. The tailored advertisement deliverysystem 200 may continuously measure the gaze of the user in response todisplaying the visual content. In some embodiments, the tailoredadvertisement delivery system 200 measures the gaze of the user via oneor more components of the user device. In some embodiments, the tailoredadvertisement delivery system 200 may measure gaze of the user based onthe device measuring the coordinates of the mixed reality device andcomparing the coordinates to the orientation of the one or more objectsin the mixed reality environment.

As shown in block 1210, the system detects one or more gestures of theuser. In some embodiments, the tailored advertisement delivery system200 continuously monitors gestures of the user via one or morecomponents of the user device to detect any movement in response todisplaying the visual content.

As shown in block 1215, the system identifies one or more expressions ofthe user. In some embodiments, the tailored advertisement deliverysystem 200 monitors expressions of the user via one or more componentsof the user device to detect any changes in response to displaying thevisual content.

In some embodiments, depending on the type of the mixed realityenvironment and the type of the user device used to access the mixedreality environment, the tailored advertisement delivery system 200 mayperform the steps described in block 1205, 1210, and 1215. For example,if the mixed reality environment is a virtual reality environment, thetailored advertisement delivery system 200 may determine the gaze of theuser and the one or more gestures of the user within the virtualenvironment. In such embodiments, the tailored advertisement deliverysystem 200 may continuously track the movements of the user in thevirtual reality environment and/or tracking the operation of a controlunit associated with the virtual reality device to determine the gazeand the one or more gestures of the user. In another example, if themixed reality environment is an augmented reality environment and theuser is experiencing augmented reality via a front facing camera of theuser device, the tailored advertisement delivery system 200 monitors anddetermines the gaze, the one or more gestures, and the one or moreexpressions of the user. In yet another example, if the mixed realityenvironment is an augmented reality environment and the user isexperiencing augmented reality via a rear facing camera of the userdevice, the tailored advertisement delivery system 200 monitors anddetermines the gaze and the one or more gestures of the user.

As shown in block 1220, the system determines that the user isinteracting with the visual content based on matching at least one ofthe (i) the gaze of the user, (ii) the one or more gestures of the user,and (iii) the one or more expressions of the user with the interactiongestures. For example, the system may identify that the user has usedswipe motion to access the visual content and may determine that theuser is interacting with the visual content. In some embodiments, thesystem determines that the user is interacting with the visual contentbased on one or more audio inputs received from the user.

As shown in block 1225, the system presents additional visual contentbased on the interaction of the user. For example, the system mayidentify that the user is interacting with the 3D visual content and maypresent a website link to the user. The user may click on the websitelink to view additional information associated with the visual content.

FIG. 13 provides a flowchart illustrating a process flow 1300 forcollecting and storing interaction metrics associated with theinteraction of the user with the advertisement, in accordance with anembodiment of the invention.

As shown in block 1305, the system continuously tracks gaze of the user.The system tracks the gaze of the user via one or more components of theuser device. For example, the system may identify that the gazing at aportion of a 2D advertisement based on continuously tracking the gaze ofthe user.

As shown in block 1310, the system generates a heat map based oncontinuously tracking the gaze of the user. The system in response tocontinuously tracking the gaze of the user, generates a heat map basedon identifying that the user is gazing at a certain feature of theadvertisement displayed to the user. For example, the system mayidentify that the user is gazing at an animated feature of the ad andmay draw a heat map around the animated feature. Using this information,the tailored advertisement delivery system 200 may determine that theuser is responding to animated visual effects and may store thisinformation in a profile of the user. In another example, the system mayidentify that the user is gazing at mint flavor among one or moreflavors of soft drinks provided in an advertisement and may draw a heatmap around the mint flavored soft drink. Heat maps may allow thetailored advertisement delivery system 200 to identify what features theuser is responding to. In some embodiments, heat maps may allow thetailored advertisement delivery system 200 to determine whether the userlikes or dislikes a product presented in the advertisement. The systemmay store such information in the profile of the user and may use thisinformation while generating future tailored advertisements to displayto the user.

In an embodiment, where the user is experiencing virtual reality, theprocess flow associated with collecting and storing metrics ends atblock 1310. In an embodiment where the user is experiencing augmentedreality using a rear facing camera of the computing device system 400,the process flow associated with collecting and storing metrics ends atblock 1310. In other embodiments where the user is experiencingaugmented reality using front facing camera of the computing devicesystem 400 or any mixed reality devices 300, the process flow associatedwith collecting and storing metrics continues to block 1315 and ends atblock 1325.

As shown in block 1315, the system continuously tracks expressions ofthe user. The system tracks the expressions of the user via one or morecomponents of the user device. As shown in block 1320, the systemdetermine emotions associated with the expressions of the user. Neuralnetworks are trained by the tailored advertisement delivery system 200to develop machine learning models that identify expressions of the userwith a confidence score. The neural networks are trained by the tailoredadvertisement delivery system 200 using one or more visual images. Insome embodiments, in response to determining that the user isinteracting with the visual content, the tailored advertisement deliverysystem 200 may analyze the expressions of the user via the one or morecomponents of the user device and the tailored advertisement deliveryapplication present in the user device. In some embodiments, thetailored advertisement delivery application present in the user devicemay transmit live feed to the tailored advertisement delivery system 200and the machine learning models present in the tailored advertisementdelivery system 200 identify the one or more expressions of the user.For example, based on the expressions of the user, the system mayidentify that the user is not happy after viewing visual content (e.g.,shade of lipstick, piece of clothing, a flavor of a soft drink, a gamingcharacter, or the like) presented to the user with 98% confidence score.

As shown in block 1325, the system stores the heat map and emotions inthe profile of the user. The system may use the heat map and theemotions in the profile of the user while generating visual content forfuture advertisements. As shown in block 1330, the system anonymizes theheat map and emotions of the user and transmits the heat map andemotions of the user to third party entities. The system may anonymizethe data and may provide the information via a dashboard (e.g., userinterface of the bidding application 260) to the third party entities.For example, the system may share that a group of users presented astrong liking towards a particular flavor of soft drink. This anonymizedinformation may be utilized by the tailored advertisement deliverysystem 200 to charge the third party entities.

As will be appreciated by one of skill in the art, the present inventionmay be embodied as a method (including, for example, acomputer-implemented process, a business process, and/or any otherprocess), apparatus (including, for example, a system, machine, device,computer program product, and/or the like), or a combination of theforegoing. Accordingly, embodiments of the present invention may takethe form of an entirely hardware embodiment, an entirely softwareembodiment (including firmware, resident software, micro-code, and thelike), or an embodiment combining software and hardware aspects that maygenerally be referred to herein as a “system.” Furthermore, embodimentsof the present invention may take the form of a computer program producton a computer-readable medium having computer-executable program codeembodied in the medium.

Any suitable transitory or non-transitory computer readable medium maybe utilized. The computer readable medium may be, for example but notlimited to, an electronic, magnetic, optical, electromagnetic, infrared,or semiconductor system, apparatus, or device. More specific examples ofthe computer readable medium include, but are not limited to, thefollowing: an electrical connection having one or more wires; a tangiblestorage medium such as a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), a compact discread-only memory (CD-ROM), or other optical or magnetic storage device.

In the context of this document, a computer readable medium may be anymedium that can contain, store, communicate, or transport the programfor use by or in connection with the instruction execution system,apparatus, or device. The computer usable program code may betransmitted using any appropriate medium, including but not limited tothe Internet, wireline, optical fiber cable, radio frequency (RF)signals, or other mediums.

Computer-executable program code for carrying out operations ofembodiments of the present invention may be written in an objectoriented, scripted or unscripted programming language such as Java,Perl, Smalltalk, C++, or the like. However, the computer program codefor carrying out operations of embodiments of the present invention mayalso be written in conventional procedural programming languages, suchas the “C” programming language or similar programming languages.

Embodiments of the present invention are described above with referenceto flowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products. It will be understood thateach block of the flowchart illustrations and/or block diagrams, and/orcombinations of blocks in the flowchart illustrations and/or blockdiagrams, can be implemented by computer-executable program codeportions. These computer-executable program code portions may beprovided to a processor of a general purpose computer, special purposecomputer, or other programmable data processing apparatus to produce aparticular machine, such that the code portions, which execute via theprocessor of the computer or other programmable data processingapparatus, create mechanisms for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

These computer-executable program code portions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the code portions stored in the computer readablememory produce an article of manufacture including instructionmechanisms which implement the function/act specified in the flowchartand/or block diagram block(s).

The computer-executable program code may also be loaded onto a computeror other programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer-implemented process such that the codeportions which execute on the computer or other programmable apparatusprovide steps for implementing the functions/acts specified in theflowchart and/or block diagram block(s). Alternatively, computer programimplemented steps or acts may be combined with operator or humanimplemented steps or acts in order to carry out an embodiment of theinvention.

As the phrase is used herein, a processor may be “configured to” performa certain function in a variety of ways, including, for example, byhaving one or more general-purpose circuits perform the function byexecuting particular computer-executable program code embodied incomputer-readable medium, and/or by having one or moreapplication-specific circuits perform the function.

Embodiments of the present invention are described above with referenceto flowcharts and/or block diagrams. It will be understood that steps ofthe processes described herein may be performed in orders different thanthose illustrated in the flowcharts. In other words, the processesrepresented by the blocks of a flowchart may, in some embodiments, be inperformed in an order other that the order illustrated, may be combinedor divided, or may be performed simultaneously. It will also beunderstood that the blocks of the block diagrams illustrated, in someembodiments, merely conceptual delineations between systems and one ormore of the systems illustrated by a block in the block diagrams may becombined or share hardware and/or software with another one or more ofthe systems illustrated by a block in the block diagrams. Likewise, adevice, system, apparatus, and/or the like may be made up of one or moredevices, systems, apparatuses, and/or the like. For example, where aprocessor is illustrated or described herein, the processor may be madeup of a plurality of microprocessors or other processing devices whichmay or may not be coupled to one another. Likewise, where a memory isillustrated or described herein, the memory may be made up of aplurality of memory devices which may or may not be coupled to oneanother.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of, and not restrictive on, the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other changes,combinations, omissions, modifications and substitutions, in addition tothose set forth in the above paragraphs, are possible. Those skilled inthe art will appreciate that various adaptations and modifications ofthe just described embodiments can be configured without departing fromthe scope and spirit of the invention. Therefore, it is to be understoodthat, within the scope of the appended claims, the invention may bepracticed other than as specifically described herein.

1-20. (canceled)
 21. A system for delivering tailored visual content inan augmented reality environment, the system comprising: at least onenetwork communication interface; at least one processing device; and atleast one non-transitory storage device with computer-readable programcode stored thereon and accessible by the at least one processingdevice, wherein the computer-readable code when executed is configuredto cause the at least one processing device to: determine that a user isaccessing an augmented reality application, via a user device; transmita first set of control signals to cause one or more components of theuser device to capture real-time environment data associated with areal-time environment of the user; identify from the real-timeenvironment data output by the one or more components of the userdevice, via neural network models, one or more bodily features of theuser captured by the one or more components of the user device; generatetailored visual content associated with at least one of the one or morebodily features captured by the one or more components of the userdevice, wherein the tailored visual content is application specificvisual content generated based on an identified type of the augmentedreality application; transmit the tailored visual content to the userdevice; and transmit a second set of control signals to cause the userdevice to display, via the augmented reality application, the tailoredvisual content associated with the one or more bodily features of theuser in an augmented reality environment of the user.
 22. The system ofclaim 21, wherein the non-transitory storage device further comprisescomputer-readable program code configured to cause the at least oneprocessing device to identify the one or more bodily features based on:generating, via the neural network models, a mesh associated with theone or more bodily features of the user captured by the one or morecomponents of the user device, wherein the mesh comprises one or moresegments; and determining, via the neural network models, a type of theone or more bodily features captured by the one or more components ofthe user device based on the generated mesh.
 23. The system of claim 22,wherein the non-transitory storage device further comprisescomputer-readable program code configured to cause the at least oneprocessing device to: calculate positional coordinates of the one ormore segments of the mesh associated with the one or more bodilyfeatures on a display of the user device for outputting the tailoredvisual content associated with the one or more bodily features; andoutput for display, the tailored visual content in the augmented realityenvironment of the user, by placing the tailored visual content in thepositional coordinates.
 24. The system of claim 23, wherein thenon-transitory storage device further comprises computer-readableprogram code configured to cause the at least one processing device toalter the tailored visual content based on the positional coordinates tooverlap the one or more bodily features.
 25. The system of claim 21,wherein the non-transitory storage device further comprisescomputer-readable program code configured to cause the at least oneprocessing device to generate the tailored visual content based on oneor more factors comprising at least one of user preferences, historicaldata, and user profile.
 26. The system of claim 21, wherein the one ormore components used to capture the real-time environment data compriseat least a front facing camera.
 27. The system of claim 21, wherein thetailored visual content is interstitial visual content.
 28. A computerprogram product for delivering tailored visual content in an augmentedreality environment, the computer program product comprising anon-transitory computer-readable storage medium comprising computerexecutable instructions for causing a computer processor to perform thesteps of: determining that a user is accessing an augmented realityapplication, via a user device; transmitting a first set of controlsignals to cause one or more components of the user device to capturereal-time environment data associated with a real-time environment ofthe user; identifying from the real-time environment data output by theone or more components of the user device, via neural network models,one or more bodily features of the user captured by the one or morecomponents of the user device; generating tailored visual contentassociated with at least one of the one or more bodily features capturedby the one or more components of the user device, wherein the tailoredvisual content is application specific visual content generated based onan identified type of the augmented reality application; transmittingthe tailored visual content to the user device; and transmitting asecond set of control signals to cause the user device to display, viathe augmented reality application, the tailored visual contentassociated with the one or more bodily features of the user in anaugmented reality environment of the user.
 29. The computer programproduct of claim 28, wherein identifying the one or more bodily featurescomprises: generating, via the neural network models, a mesh associatedwith the one or more bodily features of the user captured by the one ormore components of the user device, wherein the mesh comprises one ormore segments; and determining, via the neural network models, a type ofthe one or more bodily features captured by the one or more componentsof the user device based on the generated mesh.
 30. The computer programproduct of claim 29, wherein the computer executable instructions causethe computer processor to perform the steps of: calculating positionalcoordinates of the one or more segments of the mesh associated with theone or more bodily features on a display of the user device foroutputting the tailored visual content associated with the one or morebodily features; and outputting for display, the tailored visual contentin the augmented reality environment of the user, by placing thetailored visual content in the positional coordinates.
 31. The computerprogram product of claim 30, wherein the computer executableinstructions cause the computer processor to perform the step ofaltering the tailored visual content based on the positional coordinatesto overlap the one or more bodily features.
 32. The computer programproduct of claim 28, wherein generating the tailored visual content isbased on one or more factors comprising at least one of userpreferences, historical data, and user profile.
 33. The computer programproduct of claim 28, wherein the one or more components used to capturethe real-time environment data comprise at least a front facing camera.34. The computer program product of claim 28, wherein the tailoredvisual content is interstitial visual content.
 35. A computerimplemented method for delivering tailored visual content in anaugmented reality environment, wherein the method comprises: determiningthat a user is accessing an augmented reality application, via a userdevice; transmitting a first set of control signals to cause one or morecomponents of the user device to capture real-time environment dataassociated with a real-time environment of the user; identifying fromthe real-time environment data output by the one or more components ofthe user device, via neural network models, one or more bodily featuresof the user captured by the one or more components of the user device;generating tailored visual content associated with at least one of theone or more bodily features captured by the one or more components ofthe user device, wherein the tailored visual content is applicationspecific visual content generated based on an identified type of theaugmented reality application; transmitting the tailored visual contentto the user device; and transmitting a second set of control signals tocause the user device to display, via the augmented reality application,the tailored visual content associated with the one or more bodilyfeatures of the user in an augmented reality environment of the user.36. The computer implemented method of claim 35, wherein identifying theone or more bodily features comprises: generating, via the neuralnetwork models, a mesh associated with the one or more bodily featuresof the user captured by the one or more components of the user device,wherein the mesh comprises one or more segments; and determining, viathe neural network models, a type of the one or more bodily featurescaptured by the one or more components of the user device based on thegenerated mesh.
 37. The computer implemented method of claim 36, whereinthe method further comprises: calculating positional coordinates of theone or more segments of the mesh associated with the one or more bodilyfeatures on a display of the user device for outputting the tailoredvisual content associated with the one or more bodily features; andoutputting for display, the tailored visual content in the augmentedreality environment of the user, by placing the tailored visual contentin the positional coordinates.
 38. The computer implemented method ofclaim 37, wherein the method comprises altering the tailored visualcontent based on the positional coordinates to overlap the one or morebodily features.
 39. The computer implemented method of claim 35,wherein the method comprises generating the tailored visual contentbased on one or more factors comprising at least one of userpreferences, historical data, and user profile.
 40. The computerimplemented method of claim 35, wherein the tailored visual content isinterstitial visual content.